Combining recognition and geometry for data-driven 3D reconstruction

Thesis (S.M.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2013.Cataloged from PDF version of thesis.Includes bibliographical references (p. 47-50).Today's multi-view 3D reconstruction techniques rely almost exclusively on depth cues that come from multiple view geometry. While these cues can be used to produce highly accurate reconstructions, the resulting point clouds are often noisy and incomplete. Due to these issues, it may also be difficult to answer higher-level questions about the geometry, such as whether two surfaces meet at a right angle or whether a surface is planar. Furthermore, state-of-the-art reconstruction techniques generally cannot learn from training data, so having the ground-truth geometry for one scene does not aid in reconstructing similar scenes. In this work, we make two contributions toward data-driven 3D reconstruction. First, we present a dataset containing hundreds of RGBD videos that can be used as a source of training data for reconstruction algorithms. Second, we introduce the concept of the Shape Anchor, a region for which the combination of recognition and multiple view geometry allows us to accurately predict the latent, dense point cloud. We propose a technique to detect these regions and to predict their shapes, and we demonstrate it on our dataset.by Andrew Owens.S.M

SUN3D: A Database of Big Spaces Reconstructed Using SfM and Object Labels

Existing scene understanding datasets contain only a limited set of views of a place, and they lack representations of complete 3D spaces. In this paper, we introduce SUN3D, a large-scale RGB-D video database with camera pose and object labels, capturing the full 3D extent of many places. The tasks that go into constructing such a dataset are difficult in isolation -- hand-labeling videos is painstaking, and structure from motion (SfM) is unreliable for large spaces. But if we combine them together, we make the dataset construction task much easier. First, we introduce an intuitive labeling tool that uses a partial reconstruction to propagate labels from one frame to another. Then we use the object labels to fix errors in the reconstruction. For this, we introduce a generalization of bundle adjustment that incorporates object-to-object correspondences. This algorithm works by constraining points for the same object from different frames to lie inside a fixed-size bounding box, parameterized by its rotation and translation. The SUN3D database, the source code for the generalized bundle adjustment, and the web-based 3D annotation tool are all available at http://sun3d.cs.princeton.edu.American Society for Engineering Education. National Defense Science and Engineering Graduate FellowshipUnited States. Office of Naval Research. Multidisciplinary University Research Initiative (N000141010933

Shape Anchors for Data-Driven Multi-view Reconstruction

We present a data-driven method for building dense 3D reconstructions using a combination of recognition and multi-view cues. Our approach is based on the idea that there are image patches that are so distinctive that we can accurately estimate their latent 3D shapes solely using recognition. We call these patches shape anchors, and we use them as the basis of a multi-view reconstruction system that transfers dense, complex geometry between scenes. We "anchor" our 3D interpretation from these patches, using them to predict geometry for parts of the scene that are relatively ambiguous. The resulting algorithm produces dense reconstructions from stereo point clouds that are sparse and noisy, and we demonstrate it on a challenging dataset of real-world, indoor scenes.American Society for Engineering Education. National Defense Science and Engineering Graduate FellowshipUnited States. Office of Naval Research. Multidisciplinary University Research Initiative (N000141010933)National Science Foundation (U.S.) (Grant CGV-1212928

Shape-independent hardness estimation using deep learning and a GelSight tactile sensor

Hardness is among the most important attributes of an object that humans learn about through touch. However, approaches for robots to estimate hardness are limited, due to the lack of information provided by current tactile sensors. In this work, we address these limitations by introducing a novel method for hardness estimation, based on the GelSight tactile sensor, and the method does not require accurate control of contact conditions or the shape of objects. A GelSight has a soft contact interface, and provides high resolution tactile images of contact geometry, as well as contact force and slip conditions. In this paper, we try to use the sensor to measure hardness of objects with multiple shapes, under a loosely controlled contact condition. The contact is made manually or by a robot hand, while the force and trajectory are unknown and uneven. We analyze the data using a deep constitutional (and recurrent) neural network. Experiments show that the neural net model can estimate the hardness of objects with different shapes and hardness ranging from 8 to 87 in Shore 00 scale

Biomass production of herbaceous energy crops in the United States: field trial results and yield potential maps from the multiyear regional feedstock partnership

Current knowledge of yield potential and best agronomic management practices for perennial bioenergy grasses is primarily derived from small-scale and short-term studies, yet these studies inform policy at the national scale. In an effort to learn more about how bioenergy grasses perform across multiple locations and years, the U.S. Department of Energy (US DOE)/Sun Grant Initiative Regional Feedstock Partnership was initiated in 2008. The objectives of the Feedstock Partnership were to (1) provide a wide range of information for feedstock selection (species choice) and management practice options for a variety of regions and (2) develop national maps of potential feedstock yield for each of the herbaceous species evaluated. The Feedstock Partnership expands our previous understanding of the bioenergy potential of switchgrass, Miscanthus, sorghum, energycane, and prairie mixtures on Conservation Reserve Program land by conducting long-term, replicated trials of each species at diverse environments in the U.S. Trials were initiated between 2008 and 2010 and completed between 2012 and 2015 depending on species. Field-scale plots were utilized for switchgrass and Conservation Reserve Program trials to use traditional agricultural machinery. This is important as we know that the smaller scale studies often overestimated yield potential of some of these species. Insufficient vegetative propagules of energycane and Miscanthus prohibited farm-scale trials of these species. The Feedstock Partnership studies also confirmed that environmental differences across years and across sites had a large impact on biomass production. Nitrogen application had variable effects across feedstocks, but some nitrogen fertilizer generally had a positive effect. National yield potential maps were developed using PRISM-ELM for each species in the Feedstock Partnership. This manuscript, with the accompanying supplemental data, will be useful in making decisions about feedstock selection as well as agronomic practices across a wide region of the country

A communal catalogue reveals Earth’s multiscale microbial diversity

Our growing awareness of the microbial world’s importance and diversity contrasts starkly with our limited understanding of its fundamental structure. Despite recent advances in DNA sequencing, a lack of standardized protocols and common analytical frameworks impedes comparisons among studies, hindering the development of global inferences about microbial life on Earth. Here we present a meta-analysis of microbial community samples collected by hundreds of researchers for the Earth Microbiome Project. Coordinated protocols and new analytical methods, particularly the use of exact sequences instead of clustered operational taxonomic units, enable bacterial and archaeal ribosomal RNA gene sequences to be followed across multiple studies and allow us to explore patterns of diversity at an unprecedented scale. The result is both a reference database giving global context to DNA sequence data and a framework for incorporating data from future studies, fostering increasingly complete characterization of Earth’s microbial diversity

Safety, immunogenicity, and reactogenicity of BNT162b2 and mRNA-1273 COVID-19 vaccines given as fourth-dose boosters following two doses of ChAdOx1 nCoV-19 or BNT162b2 and a third dose of BNT162b2 (COV-BOOST): a multicentre, blinded, phase 2, randomised trial

Background Some high-income countries have deployed fourth doses of COVID-19 vaccines, but the clinical need, effectiveness, timing, and dose of a fourth dose remain uncertain. We aimed to investigate the safety, reactogenicity, and immunogenicity of fourth-dose boosters against COVID-19.Methods The COV-BOOST trial is a multicentre, blinded, phase 2, randomised controlled trial of seven COVID-19 vaccines given as third-dose boosters at 18 sites in the UK. This sub-study enrolled participants who had received BNT162b2 (Pfizer-BioNTech) as their third dose in COV-BOOST and randomly assigned them (1:1) to receive a fourth dose of either BNT162b2 (30 µg in 0·30 mL; full dose) or mRNA-1273 (Moderna; 50 µg in 0·25 mL; half dose) via intramuscular injection into the upper arm. The computer-generated randomisation list was created by the study statisticians with random block sizes of two or four. Participants and all study staff not delivering the vaccines were masked to treatment allocation. The coprimary outcomes were safety and reactogenicity, and immunogenicity (antispike protein IgG titres by ELISA and cellular immune response by ELISpot). We compared immunogenicity at 28 days after the third dose versus 14 days after the fourth dose and at day 0 versus day 14 relative to the fourth dose. Safety and reactogenicity were assessed in the per-protocol population, which comprised all participants who received a fourth-dose booster regardless of their SARS-CoV-2 serostatus. Immunogenicity was primarily analysed in a modified intention-to-treat population comprising seronegative participants who had received a fourth-dose booster and had available endpoint data. This trial is registered with ISRCTN, 73765130, and is ongoing.Findings Between Jan 11 and Jan 25, 2022, 166 participants were screened, randomly assigned, and received either full-dose BNT162b2 (n=83) or half-dose mRNA-1273 (n=83) as a fourth dose. The median age of these participants was 70·1 years (IQR 51·6–77·5) and 86 (52%) of 166 participants were female and 80 (48%) were male. The median interval between the third and fourth doses was 208·5 days (IQR 203·3–214·8). Pain was the most common local solicited adverse event and fatigue was the most common systemic solicited adverse event after BNT162b2 or mRNA-1273 booster doses. None of three serious adverse events reported after a fourth dose with BNT162b2 were related to the study vaccine. In the BNT162b2 group, geometric mean anti-spike protein IgG concentration at day 28 after the third dose was 23 325 ELISA laboratory units (ELU)/mL (95% CI 20 030–27 162), which increased to 37 460 ELU/mL (31 996–43 857) at day 14 after the fourth dose, representing a significant fold change (geometric mean 1·59, 95% CI 1·41–1·78). There was a significant increase in geometric mean anti-spike protein IgG concentration from 28 days after the third dose (25 317 ELU/mL, 95% CI 20 996–30 528) to 14 days after a fourth dose of mRNA-1273 (54 936 ELU/mL, 46 826–64 452), with a geometric mean fold change of 2·19 (1·90–2·52). The fold changes in anti-spike protein IgG titres from before (day 0) to after (day 14) the fourth dose were 12·19 (95% CI 10·37–14·32) and 15·90 (12·92–19·58) in the BNT162b2 and mRNA-1273 groups, respectively. T-cell responses were also boosted after the fourth dose (eg, the fold changes for the wild-type variant from before to after the fourth dose were 7·32 [95% CI 3·24–16·54] in the BNT162b2 group and 6·22 [3·90–9·92] in the mRNA-1273 group).Interpretation Fourth-dose COVID-19 mRNA booster vaccines are well tolerated and boost cellular and humoral immunity. Peak responses after the fourth dose were similar to, and possibly better than, peak responses after the third dose

Learning visual models from paired audio-visual examples

Thesis: Ph. D., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, 2016.Cataloged from PDF version of thesis.Includes bibliographical references (pages 93-104).From the clink of a mug placed onto a saucer to the bustle of a busy café, our days are filled with visual experiences that are accompanied by distinctive sounds. In this thesis, we show that these sounds can provide a rich training signal for learning visual models. First, we propose the task of predicting the sound that an object makes when struck as a way of studying physical interactions within a visual scene. We demonstrate this idea by training an algorithm to produce plausible soundtracks for videos in which people hit and scratch objects with a drumstick. Then, with human studies and automated evaluations on recognition tasks, we verify that the sounds produced by the algorithm convey information about actions and material properties. Second, we show that ambient audio - e.g., crashing waves, people speaking in a crowd - can also be used to learn visual models. We train a convolutional neural network to predict a statistical summary of the sounds that occur within a scene, and we demonstrate that the visual representation learned by the model conveys information about objects and scenes.by Andrew Owens.Ph. D

Biomass production of herbaceous energy crops in the United States: field trial results and yield potential maps from the multiyear regional feedstock partnership

Current knowledge of yield potential and best agronomic management practices for perennial bioenergy grasses is primarily derived from small-scale and short-term studies, yet these studies inform policy at the national scale. In an effort to learn more about how bioenergy grasses perform across multiple locations and years, the U.S. Department of Energy (US DOE)/Sun Grant Initiative Regional Feedstock Partnership was initiated in 2008. The objectives of the Feedstock Partnership were to (1) provide a wide range of information for feedstock selection (species choice) and management practice options for a variety of regions and (2) develop national maps of potential feedstock yield for each of the herbaceous species evaluated. The Feedstock Partnership expands our previous understanding of the bioenergy potential of switchgrass, Miscanthus, sorghum, energycane, and prairie mixtures on Conservation Reserve Program land by conducting long-term, replicated trials of each species at diverse environments in the U.S. Trials were initiated between 2008 and 2010 and completed between 2012 and 2015 depending on species. Field-scale plots were utilized for switchgrass and Conservation Reserve Program trials to use traditional agricultural machinery. This is important as we know that the smaller scale studies often overestimated yield potential of some of these species. Insufficient vegetative propagules of energycane and Miscanthus prohibited farm-scale trials of these species. The Feedstock Partnership studies also confirmed that environmental differences across years and across sites had a large impact on biomass production. Nitrogen application had variable effects across feedstocks, but some nitrogen fertilizer generally had a positive effect. National yield potential maps were developed using PRISM-ELM for each species in the Feedstock Partnership. This manuscript, with the accompanying supplemental data, will be useful in making decisions about feedstock selection as well as agronomic practices across a wide region of the country